Papers with Llama-family models
HIGGS: Pushing the Limits of Large Language Model Quantization via the Linearity Theorem (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing large language models focus on breaking down the problem into layer-wise sub-problems and minimizing per-layer error, but this approach lacks theoretical justification and the metrics employed may be sub-optimal. |
| Approach: | They propose a "linearity theorem" establishing a direct relationship between the layer-wise reconstruction error and the model perplexity increase due to quantization. |
| Outcome: | The proposed method outperforms previous data-free methods and improves accuracy-compression trade-offs on Llama-family models. |
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference (2025.acl-long)
Copied to clipboard
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, Yaodong Yang
| Challenge: | Using large-scale annotation data, large language models can generate noise, errors and biases, leading to unexpected behaviours. |
| Approach: | They propose a dataset to promote safety alignment in large language models . they separate helpfulness and harmlessness annotations for question-answering pairs . |
| Outcome: | The proposed dataset provides 44.6k prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels, with answers generated by Llama-family models. |
Efficient Domain Continual pretraining by Mitigating the Stability Gap (2025.acl-long)
Copied to clipboard
| Challenge: | Continual pretraining is an important approach for Large Language Models to improve their performance in target domains, learn new topics and languages, and even boost their general capabilities. |
| Approach: | They propose a training strategy that mitigates instability by increasing the number of epochs, along with two data sampling strategies targeting data domain relevance and corpus distribution. |
| Outcome: | The proposed training strategy improves the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% using only 40% of the original training budget, while also enhancing general task performance without causing forgetting. |